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intaining  the  1200  PSI  Steam  Propulsion  Plant.  This  report  describes 
the  AN/BQR-20A  application  in  which  Sonar  Technician  Teams  from  12  FBM 
submarines  were  given  a diagnostic  pretest  and  then  retested  approximately 
5 months  later.  STs  from  four  submarines  were  assigned  to  each  of  three 
experimental  groups:  (1)  a Control  Group  in  which  the  participants  were 

given  feedback  on  the  pretest  in  terms  of  an  overall  percentage  score, 

(2)  a Feedback  Group  in  which  the  members  were  given  both  an  overall  per- 
centage score  and  a written  outline  indicating  their  specific  weaknesses, 
and  (3)  a Feedback  + Training  Group  in  which  the  members  were  given  the 
same  type  of  information  as  the  Feedback  Group  but  were  assigned  specific 
remedial  training.  Diagnostic  testing/shipboard  remedial  training  appeared 
to  be  successful  ijp,  both  detecting  and  overcoming  deficiencies  of  Fleet 
personnel.  Feedback  alone  was  not  sufficient  to  bring  about  improvement 
in  job  performance. 
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series  of  reports  relating  to  Subproject  Z0108-PN.24,  Personnel  Readiness 
Training.  The  first  report,  NPRDC  Special  Report  75-8,  A Personnel  Readiness 
Training;  Program:  Initial  Project  Developments,  provided  an  overview  of 

the  project.  Subsequent  reports  will  describe  the  Personnel  Readiness 
Training  Programs  for  Missile  Technicians  and  Boiler  Technicians.  A final 
report  will  summarize  findings  and  conclusions  across  all  three  applications. 
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SUMMARY 


Problem 


{ 


- Personnel  Readiness  Training  Program  is  concerned  with  the  feasibility 

of  utilizing  a diagnostic  testing/shipboard  remedial  training  system  to 
improve  the  readiness  levels  of  Fleet  personnel.  In  such  a system,  per- 
formance-oriented tests,  which  focus  on  the  essential  skills  needed  to  do  a 
job  and  refer  to  knowledges  only  as  they  are  necessary  to  support  those 
skills,  are  used  to  diagnose  deficiencies  in  job  performance.  Self-instruc- 
tiona l training  materials  which  relate  to  essential  aspects  of  the  job  are 
individually  prescribed  to  correct  deficiencies  revealed  by  the  diagnostic 
tests.  To  obtain  information  on  how  and  where  such  a system  might  work 
testing  and  training  programs  were  developed  for  three  applications:  (1) 

the  submarine  Sonar  Technician  (ST)  operating  the  AN/BQR-20A,  (2)  the  sub- 
marine Missile  Technician  (MT)  maintaining  the  Missile  Test  and  Readiness 
•.quipment  (MTRF) , and  (3)  the  Boiler  Technician  (BT)  operating  and  main- 
taining the  1200  PSI  Steam  Propulsion  Plant. 


Purpose 


The  purpose  of  the  effort  described  in  this  report  was  to  determine 

ether  the  testing  and  training  programs  developed  for  the  ST  application 

!»/Lin!!rUTKntal  in  imProvln8  the  performance  of  STs  who  operate  the 
AN/BQR-20A. 


Approach 


Sonar  Technician  teams  from  12  Fleet  Ballistic  Missile  (FBM)  submarines 
were  given  a diagnostic  pretest  and  then  retested  approximately  5 months 
later.  The  test  items  concerned  both  the  knowledge  considered  essential 
for  proper  operation  of  the  AN/BQR-20A  and  the  skills  required  to  set  up 
the  front  panel  controls  for  various  tactical  situations.  Three  experi- 
mental groups,  each  consisting  of  ST  teams  from  four  submarines,  were  used 
In  the  evaluation:  a Control  Group,  a Diagnostic  Feedback  Group,  and  a 

Diagnostic  Feedback  + Training  Group.  Following  the  pretest,  members  of 
the  Control  Group  were  given  feedback  on  their  test  performance  in  terms 
of  an  overall  percentage  score.  Members  of  the  Feedback  Group  were  given 
overall  percentage  scores,  but,  in  addition,  were  provided  with  a written 
outline  indicating  their  Individual  weaknesses.  Members  of  the  Diagnostic 
Feedback  + Training  Group  were  provided  with  the  same  typo  of  information 
as  tlie  feedback  Group,  but,  additionally,  were  assigned  specific  remedial 
training  materials  covering  their  individual  weaknesses. 


Results 


In  general,  on  both  the  performance  test  and  the  equipment-related 
questions  on  the  written  test,  the  Training  Group  showed  significant  improve- 
ment from  pre-  to  posttest  while  the  Feedback  and  Control  Groups  showed 
little  or  no  improvement. 
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Conclusions 
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\ 


1 • ^or  operation  of  the  AN/BQR— 20A,  the  diagnostic  testing/shipboard 
training  techniques  which  were  developed  are  effective  in  both  detecting 
and  overcoming  deficiencies. 

2.  In  the  ST  application,  feedback  in  the  form  of  specific  performance 
deficiencies  was  not  in  itself  sufficient  to  bring  about  improvements  in 
Job  performance. 

3.  Sonar  Technicans  in  the  FBM  Fleet  appear  to  have  deficiencies  in 
skills  and  knowledges  related  to  the  operation  of  the  AN/BQR-2QA. 

Recommendat ion 

In  order  to  improve  the  proficiency  level  of  Fleet  operators,  it  is 
recommended  that  the  AN/BQR— 20A  diagnostic  testing/shipboard  training  pro- 
gram be  implemented.  Implementation  should  start  with  a shipboard  training 
phase  and  diagnostic  testing  should  begin  at  a later  date. 
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INTRODUCTION 


Problem 


The  Personnel  Readiness  Training  Program  is  concerned  with  the  feasibility 
of  utilizing  a diagnostic  testing/shipboard  training  system  to  improve  the 
readiness  levels  of  Fleet  personnel.1  In  such  a system,  performance-oriented 
tests  are  used  to  diagnose  deficiencies  in  job  performance.  Self-instructional 
training  materials  that  relate  to  essential  aspects  of  the  job  are  then  in- 
dividually prescribed  to  correct  deficiencies  revealed  by  the  diagnostic 
tests. 

Background 


The  degree  to  which  critical  job  skills  can  be  improved  through  a system 
of  diagnostic  testing  and  shipboard  training  may  depend  on  the  rating  and/or 
the  type  of  task  involved.  Therefore,  to  obtain  information  on  how  and 
where  the  system  might  work,  testing  and  training  programs  were  developed 
for  three  applications.  The  ratings  and  related  equipments  selected  for 
the  program  evaluation  were:  (1)  the  submarine  Sonar  Technician  (ST) 

operating  the  AN/BQR-20A,  (2)  the  submarine  Missile  Technician  (MT)  main- 
taining the  Missile  Test  and  Readiness  Equipment  (MTRE) , and  (3)  the  Boiler 
Technician  (BT)  operating  and  maintaining  the  1200  PSI  Steam  Propulsion 
Plant.  These  ratings  and  equipments  were  chosen  because  they  were  critical 
to  the  missions  of  their  ships  and  they  contained  a sufficient  variety  of 
operator  and  maintenance  tasks  to  permit  the  results  to  be  generalized  to 
other  areas.  Additionally,  there  was  evidence  that  performance  deficiencies 
might  be  present  in  these  areas. 

Purpose 

The  purpose  of  this  effort  was  to  determine  whether  the  testing  and 
training  programs  developed  for  submarine  Sonar  Technicians  (STs)  were 
instrumental  in  improving  their  performance  on  the  operation  of  the  AN/BOR-20A. 

The  AN/BQR-20A,  a passive  sonar  receiving  set  found  on  board  all  FBM 
submarines,  is  used  for  detection  and  analysis  of  sonar  contacts.  In  ful- 
filling their  mission,  a prime  requisite  for  FBM  submarines  is  that  they 
avoid  being  detected  by  a potential  adversary.  Because  of  its  long-range 
detection  capability,  the  AN/BQR-20A  serves  a vital  role  in  the  mission  of 
these  submarines.  This  specialized  piece  of  sonar  equipment  requires  only 
a single  operator  and  has  a relatively  limited  number  of  tasks  involved  in 
its  operation.  Thus,  within  acceptable  time  limits,  it  was  possible  to  test 
all  major  operator  tasks  and  most  auxiliary  tasks.  Additionally,  since 
the  AN/BQR-20A  is  installed  at  most  submarine  shore  facilities,  testing 
could  be  accomplished  ashore  with  the  actual  equipment  during  FBM  submarine 
off-crew  periods. 


'haabs,  C.  .1.,  Main,  R.  E. , Abrams,  A.  .1.,  & Steinemann,  J.  H.  A personnel 
readiness  training  program:  Initial  project  developments  (NPRDC  Special 

Report  75-8).  San  Diego:  Navy  Personnel  Research  and  Development  Center, 

April  1975. 


METHOD 


Description  of  Materials 

AN/BQR-20A  Operator  Task  Analysis 

It  is  generally  accepted  that  training  and  testing  materials  are 
most  valid  when  they  are  derived  from  a task  analysis.  When  this  project 
began,  no  task  analysis  was  available  for  the  operation  of  the  AN/BQR-20A. 
Therefore,  initial  efforts  were  devoted  to  developing  a detailed  task 
analysis.  This  was  done  by  (1)  breaking  down  the  overall  operator  task 
into  the  specific  tactical  employments  of  the  system  (i.e.,  detection, 
contact  investigation,  tracking,  and  auxiliary  uses),  (2)  outlining,  for 
each  tactical  employment,  a step-by-step  procedure  for  properly  setting  the 
front  panel  and  cathode  ray  tube  (CRT)  controls  according  to  operational 
doctrine,  and  (1)  providing,  at  each  step  in  the  procedure,  an  explanation 
as  to  why  that  particular  control  setting  was  recommended  by  doctrine.  In 
this  task  analysis  effort,  operational  doctrines  of  Commander  Submarine 
Force,  Atlantic  (COMSUULANT)  and  Commander  Submarine  Force,  Pacific 
(COMSIJBPAC)  were  used  as  authoritative  sources.  Since  the  doctrines  were 
not  explicit  for  contact  investigation,  procedures  recommended  by  personnel 
from  the  Navy  Intelligence  Support  Center  (NISC)  were  incorporated  into 
the  task  analysis.  With  this  exception,  the  analysis  is  consistent  with 
doctrine  promulgated  by  COMSUBLANT  and  COMSUBPAC.  The  final  version  was 
reviewed  in  detail  with  experienced  STs  in  the  Fleet,  and  with  personnel 
at  NISC,  at  Submarine  Development  Group  TWO,  and  at  the  Submarine  School, 

New  London.2 

Diagnostic  Test 

The  diagnostic  test  instruments,  consisting  of  a hands-on  performance 
test  and  a written  test,  were  based  upon  the  task  analysis.  For  the  perfor- 
mance test,  the  AN/B0R-20A  was  interfaced  with  an  AN/UNQ-7  tape  recorder, 
and  tapes  were  used  to  provide  signal  input  into  the  system.  Four  tapes 
were  developed  for  this  test;  three  contained  simulated  sonar  signals, 
produced  by  using  signal  generators,  and  one  contained  active  sonar  trans- 
missions. In  most  instances,  simulated  materials  were  used  to  ensure  that 
an  ST  would  see  a clear  presentation  on  the  CRT  if  he  had  the  equipment 
set  up  properly  for  a given  problem.  Prior  to  administration  of  the  per- 
formance test,  the  ST  was  given  a set  of  written  instructions  which  provided 
necessary  background  Information  and  outlined  the  tasks  he  was  expected 
to  perform.  The  test  required  each  ST  to  set  up  the  equipment  and  to 
perform  the  search  task,  the  contact  investigation  task,  and  the  tracking 
task  on  three  separate  tapes.  On  two  of  these  tapes,  he  was  also  required 
to  set  up  the  equipment  to  perform  a signal-to-noise  ratio  (SNR)  calcula- 
tion. On  the  fourth  tape,  he  was  required  to  set  up  the  equipment  for 
a ping  interception.  In  summary,  then,  each  ST  was  required  to  perform 
tiie  following  tasks:  three  search,  three  contact  investigation,  three 

tracking,  two  SNR  calculations,  and  one  ping  interception. 


2 This  task  analysis  is  a classified  document.  Copies  are  available  to 
authorized  personnel  upon  request  to  this  Command.  Requests  should  be 
addressed  to  the  attention  of  Code  302. 


In  the  performance  test,  the  emphasis  was  on  accuracy  in  selecting 
those  front  panel  anti  CRT  control  settings  which  were  recommended  by  opera- 
tional doctrine  for  given  tactical  situations.  Thus,  all  pertinent  front 
panel  and  CRT  controls  had  to  be  properly  set  for  each  problem.  Those  con- 
trols on  which  the  various  settings  have  little  effect  on  system  performance 
(e.g.,  slope  control)  were  not  considered  in  scoring.  Also,  for  controls 
on  which  a particular  setting  may  be  partially  a matter  of  personal  preference 
(e.g.,  CRT  brightness  control),  any  selected  setting  was  scored  as  correct 
if  it  fell  within  an  acceptable  range.  In  the  case  of  the  CRT  brightness 
control,  a given  setting  was  scored  as  incorrect  only  if  it  was  so  bright 
or  so  dim  that  pertinent  information  might  be  lost  or  obscured  on  the  CRT. 

A highly  qualified  rater  observed  the  performance  of  each  ST  as  he  set  up 
the  equipment.  When  the  rater  was  informed  by  the  ST  that  he  had  completed 
a problem,  the  rater  recorded  the  position  of  each  pertinent  switch  on  a 
scoring  sheet  specially  designed  so  that  the  actual  scoring  could  be  done 
later.  Scoring  procedures  are  described  below: 

1.  On  the  search  task,  11  switches  were  scored  on  each  of  the  three 
problems,  resulting  in  a total  of  33  observations  per  ST. 

2.  For  the  contact  investigation  task,  the  test  instructions  pro- 
vided two  hypothetical  conditions  for  performing  the  task.  In  the  first 
condition  (applicable  to  the  first  two  problems),  the  ST  was  told  that  the 
tactical  situation  was  such  that  it  provided  ample  time  to  make  a thorough 
investigation  of  the  contact.  In  the  second  condition  (applicable  to  the 
third  problem),  the  ST  was  told  that  the  tactical  situation  provided  only 
minimum  time  to  investigate  for  additional  contact  information.  This  was 
done  since  somewhat  different  procedures  are  required  in  each  of  these  tac- 
tical situations.  On  each  of  the  first  two  problems,  11  switches  were  scored; 
on  the  third,  10  switches  were  scored,  yielding  a total  of  32  observations 
per  ST. 


3.  On  the  tracking  task,  11  switches  were  scored  on  each  of  the 
three  problems,  resulting  in  a t tal  of  33  observations  per  ST. 

4.  The  ping  interception  task  involved  one  problem  in  which  the 
ST  was  required  to  set  up  the  equipment  to  intercept  a ping  pulse  of  un- 
known frequency.  For  this  task,  seven  switches  were  scored  for  a total  of 
seven  observations  per  ST. 

5.  For  the  SNK  calculation  task,  the  ST  was  required — in  two 
problems — to  set  tip  the  equipment  to  matte  a SNR  calculation  and  then,  from 
the  CRT  presentation,  to  report  the  approximate  dB  value  of  the  ratio.  Since 
the  stress  of  the  performance  test  was  on  correct  set-up  procedures,  the  ST 
was  not  required  to  apply  correction  factors  to  determine  the  exact  dB  values. 
Inasmuch  as  this  measurement  can  be  made  at  a variety  of  front  panel  control 
settings,  only  the  mode  switch  and  the  dB  value  for  the  ratio  were  scored, 
yielding  four  observations  per  ST. 

For  each  ST,  there  were  109  operational  steps  that  were  observed  over  a 
total  of  12  problems. 


When  scoring  the  test,  the  switch  positions  recorded  on  the  answer 
sheets  were  compared  to  predetermined  standards.  The  standard  established 
for  the  individual  switches  was  the  switch  position  recommended  by  operational 
doctrine  for  the  given  tactical  situation.  Since  each  switch  setting  was 
recorded  by  the  rater  during  the  test  and  the  scoring  sheets  were  later  com- 
pared against  predetermined  standards,  subjective  judgments  on  the  part 
of  the  rater  were  largely  avoided,  both  during  the  observation  of  performance 
and  in  the  scoring  of  the  test.  As  a check  on  the  amount  of  rater  subjec- 
tivity that  might  still  be  present,  the  objectivity  of  the  performance  test 
was  evaluated  by  using  two  trained  observers  during  the  pretesting  of  ST 
teams  from  three  submarines  and  comparing  their  observations. 

The  42-item  written  test  focused  upon  specific  knowledges  considered 
to  be  essential  for  proper  operation  of  the  system.  Thirty-one  of  these 
questions  were  related  specifically  to  the  operation  of  the  AN/BQR-20A,  and 
11  were  concerned  with  general  sonar  knowledges  which  bear  on  the  proper 
utilization  of  this  equipment.  Questions  in  the  latter  category  covered 
sucli  areas  as  selection  of  appropriate  passive  sonar  under  given  conditions 
and  calculation  of  the  f igure-of-merit  when  the  values  for  the  terms  of 
the  equation  were  given.  Tn  general,  the  written  test  covered  areas  which 
were  not  covered  in  the  performance  test.  Both  the  performance  and  written 
tests  were  tried  out  on  ST  teams  from  FRM  submarines  before  data  gathering 
began. 


Remedial  Shipboard  Training  Materials 

The  shipboard  training  materials  consisted  of  six  self-instructional 
units,  five  logic  tress,  two  !4-inch  training  tapes,  and  six  drills.  In 
general,  the  self-instructional  units  presented  basic  information  on  system 
operation  and  the  recommended  settings  for,  and  the  proper  relationship 
between,  the  individual  front  panel  and  CRT  controls  for  various  tactical 
employments  of  the  system.  The  logic  trees  provided  the  trainee  with  a 
systematic  procedure  for  properly  setting  up  the  AN/BQR-20A  for  search, 
tracking,  contact  investigation,  determination  of  SNR,  and  ping  intercep- 
tion. The  two  ’.-inch  training  tapes,  which  contained  simulated  sonar 
signals,  were  used  in  conjunction  with  the  six  drills  to  provide  hands-on 
training  on  the  equipment.  The  tapes  provided  signal  input  to  the  AN/BQR-20A 
via  a tape  recorder,  so  that  the  trainee  could  observe  a presentation  on 
the  CRT  as  he  worked  through  the  drills. 

There  was  one  drill  for  each  of  the  tasks  (e.g.,  search,  track, 
contact  investigation).  The  drill  format  provided  a switch-by-switch 
procedure  for  setting  up  the  front  panel  and  CRT  controls  for  that  par- 
ticular tactical  employment  of  the  system.  For  each  switch,  the  proper 
setting  was  given  and  an  explanation  was  provided  as  to  why  that  particular 
setting,  was  recommended  for  the  given  situation.  In  a separate  column,  the 
incorrect  settings  for  each  switch  were  shown  and  an  explanation  was  given 
of  the  adverse  effects  each  incorrect  setting  could  have  on  system  perfor- 
mance. This  format  for  the  drills  enabled  one  trainee  to  act  as  an  instructor 
while  another  set  up  the  controls  on  the  equipment  for  the  given  problem. 

With  this  method,  immediate  feedback  on  any  error  in  setting  a switch  was 
given  to  the  trainee  along  with  an  explanation  as  to  why  that  incorrect 
setting  was  undesirable  in  terms  of  system  performance. 
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Shipboard  training  materials  were  assigned  on  the  basis  of  diagnostic 
test  error  patterns.  For  this  purpose,  performance  test  results  were  analyzed 
both  in  terms  of:  (1)  the  number  of  times  an  individual  switch  was  set  in- 

correctly across  all  tasks  and  (2)  the  number  of  control  panel  and  CRT  switches 
set  incorrectly  within  each  task.  Errors  on  specific  written  test  items  and 
the  number  of  times  an  individual  switch  was  set  improperly  in  the  performance 
test  were  used  as  the  basis  for  assigning  the  self-instructional  units.  The 
logic  trees  and  the  drills  were  assigned  on  the  basis  of  the  number  of 
switches  set  incorrectly  within  each  task.  Members  of  the  Training 
Group  were  asked  to  complete  their  assigned  training  materials  at  times 
convenient  for  them  during  their  subsequent  patrol.  A master  list  of  training 
material  assigned  to  each  individual  ST  was  provided  to  the  Leading  Petty 
Officer  (LPO)  for  purposes  of  monitoring  training  and  maintaining  a record 
as  each  ST  completed  the  assigned  materials.  Just  how  and  when  the  training 
was  to  be  accomplished  was  left  up  to  the  discretion  of  the  individual 
LPOs.  The  recommended  progression  in  using  the  training  materials  was  from 
self-instructional  units,  to  logic  trees,  to  drills. 

Experimental  Conditions 

The  experimental  design  shown  in  Table  1 was  used  to  evaluate  the  ST 
application  of  the  Personnel  Readiness  Training  Program.  As  depicted  in 
the  design,  three  experimental  groups,  each  consisting  of  ST  teams  from  four 
submarines,  were  used  in  the  evaluation:  A Control  Group,  a Diagnostic 

Feedback  Group,  and  a Diagnostic  Feedback  + Training  Group.  Members  of 
all  three  groups  were  given  the  previously  described  diagnostic  tests  and 
then  were  retested  approximately  5 months  later  using  the  same  tests.  Since 
the  same  tests  were  used  for  the  posttest,  no  feedback  concerning  which 
specific  written  test  items  were  missed  or  which  specific  control  settings 
were  improperly  set  was  given  during  the  pretest.  What  happened  to  the 
individuals  in  the  interim  depended  upon  the  group  to  which  they  were  assigned. 


Table  1 

Experimental  Design  Used  in  ST  Application  of  the 
Personnel  Readiness  Training  Program 


Group 

Experimental  Conditions 

Control 

Pretest  with  non- 
specific feedback 

Patrol 

Posttest 

Diagnostic 

Feedback 

Pretest  with  specific 
feedback  on  weaknesses 

Patrol 

Posttest 

Diagnostic 
Feedback  + 
Training 

Pretest  with  specific 
feedback  on  weaknesses 
and  remedial  training 
assl gned 

Patrol 

AN/BQR-20A 

Training 

Package 

onboard 

Posttest 
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Description  of  Groups 


Control  Group.  Following  the  pretest,  ST  teams  in  this  group  were 
given  feedback  on  their  test  performance  in  terms  of  an  overall  percentage 
score.  For  example,  an  ST  participating  in  this  group  might  be  told  that 
he  answered  70  percent  of  the  written  test  items  correctly  and  set  62  percent 
of  the  controls  in  the  recommended  positions  on  the  performance  test.  Members 
of  this  group  were  not  provided  with  any  information  concerning  their  specific 
deficiencies  and  they  were  not  given  suggestions  or  directions  as  to  how 
their  deficiencies  might  be  corrected. 

Diagnostic  Feedback  Group.  Following  the  pretest,  members  of  this 
group  were  also  given  overall  percentage  scores  but,  in  addition,  they  were 
provided  with  a written  outline  indicating  their  individual  weaknesses.  This 
outline  pointed  out  specific  weaknesses  both  in  terms  of  (1)  front  panel  and 
CRT  controls  which  had  frequently  been  used  improperly  and  (2)  tasks 
(e.g.,  search,  track,  contact  investigation)  which  showed  overall  perfor- 
mance deficiencies.  For  example,  in  addition  to  his  test  percentage  scores, 
an  ST  might  be  informed  that  he  was  weak  on  the  use  of  the  range  control 
switch  and  the  signal  gain  controls  and  that  his  performance  was  inadequate 
on  the  tasks  of  search  and  contact  investigation.  As  witli  the  Control 
Group,  members  of  this  group  were  not  provided  with  any  information  as  to 
how  their  deficiencies  might  be  corrected. 

Diagnostic  Feedback  + Training  Group.  Following  the  pretest,  members 
in  this  group  were  given  the  same  type  of  feedback  as  was  given  to  those  in 
the  Diagnostic  Feedback  Group,  i.e.,  percentage  scores  and  specific  weak- 
nesses in  terms  of  individual  switches  and  tasks.  In  addition,  they  were 
assigned  specific  remedial  training  materials  covering  their  individual 
weaknesses.  For  example,  if  the  test  results  for  an  ST  in  this  group  in- 
dicated that  his  performance  was  deficient  in  the  use  of  the  range  switch 
and  in  the  use  of  signal  gain  control  switches,  he  was  assigned  the  self- 
instructional  units  which  covered  the  use  of  the  range  switch  and  the  proper 
settings  for  the  gain  control  switches.  Additionally,  he  was  assigned  logic 
trees  and  drills  for  each  area  in  which  his  performance  was  deficient. 

Sample 

ST  teams  from  12  Fleet  Ballistic  Missile  (FBM)  submarines — six  Atlantic 
Fleet  (I.ANTFLT)  and  six  Pacific  Fleet  (PACFLT) — participated  in  the  program 
evaluation.  The  testing  program  was  carried  out  in  the  FBM  Fleet  because: 

(1)  the  AN/BQR-20A  is  found  onboard  all  FBM  submarines,  (2)  the  regularity 
of  the  FRM  patrol  cycle  provided  reasonable  assurance  that  the  time  frame 
between  pre-  and  posttesting  would  be  relatively  constant  across  crews,  and 
(3)  FBM  crews  could  be  tested  at  shore  facilities  during  the  off-crew  period, 
thereby  simplifying  the  problem  of  arranging  for  testing  times  and  locations. 

In  LANTFLT,  assignment  of  ST  teams  to  the  various  groups  was  made  by 
Commander,  Submarine  Group  TWO  for  the  three  submarines  tested  in  New  London 
and  Corimander,  Submarine  Group  SIX  for  the  three  submarines  tested  in 
Charleston.  In  PACFLT,  however,  the  decision  of  group  assignment  was 
left  up  to  the  research  team.  Since  pretesting  was  conducted  initially 
in  LANTFLT,  assignment  of  ST  teams  in  PACFLT  was  done  on  the  basis  of 


the  number  of  STs  in  a team  in  order  to  balance  the  number  of  participants 
in  each  experimental  group.  Of  the  69  STs  who  were  pretested,  22  were 
assigned  to  the  Control  Group,  23  to  the  Feedback  Group,  and  24  to  the 
Feedback  + Training  Group. 

A total  of  56  STs  were  included  in  the  final  sample.3  Length  of  Navy 
experience  for  STs  varied  from  a minimum  of  10  months  to  a maximum  of  18 
years,  with  the  average  being  4.8  years.  The  distribution  of  paygrades 
and  the  percentage  of  the  total  sample  each  represents  are  shown  in 
Table  2. 


Table  2 

Percentage  Distribution  of  Pay  Grades  in  Experimental  Groups 


Pay  Grades 

Diagnostic 
Feedback  + 
Training 
Group 
N=23 

Diagnostic 

Feedback 

Group 

N=18 

Control 

Group 

N=15 

Total  Sample 
N*=56 

E-2/E-3 

13 

6 

7 

9 

E-4 

40 

61 

66 

53 

E-5 

17 

22 

13 

18 

E-6 

17 

11 

7 

13 

E-7/E-8 

13 

0 

7 

7 

Procedures 


Fleet  testing  was  conducted  by  members  of  the  research  team  at  FBM  off- 
crew  ports  at  New  London,  Charleston,  and  Pearl  Harbor.  The  procedure  followed 
in  the  testing  program  was  to  initially  assemble  an  entire  ST  team  in  a 
classroom  to  take  the  written  test.  Before  the  pretest  was  administered, 
the  STs  were  asked  to  complete  a short  biographical  questionnaire  and  they 


Of  the  original  69  STs  who  took  the  pretest,  13  were  unavailable  for 
posttesting  due  to  discharage  from  the  Navy,  transfer,  or  leave.  Of  the 
56  STs  included  in  the  final  sample,  8 had  been  assigned  to  various  courses 
at  Fleet  Anti-Submarine  Warfare  Training  Center,  Pacific  (FLEASWTItACENPAC) , 
San  Diego.  Arrangements  were  made  with  FLEASWTRACENPAC  to  retest  these 
students  at  the  Center.  None  of  them  had  received  training  on  the  AN/BQR-20A 
in  their  courses  at  the  Center.  Since  statistical  tests  revealed  no  differ- 
ences between  this  student  group  and  those  remaining  in  the  submarine  crews, 
the  students  were  included  in  the  final  sample. 


wore  briefed  on  the  general  purpose  of  the  project  and  the  procedures  to  be 
followed  in  the  testing  program.  However,  they  were  not  told  that  there 
were  three  groups  involved  in  the  study.  Depending  upon  the  group  to  which 
they  were  assigned,  each  team  wis  told  the  type  of  feedback  information  they 
would  receive  following  testing.  For  example,  members  of  the  Training  Croup 
were  told  that  they  would  be  given  their  percentage  scores  on  the  tests, 
an  outline  pointing  out  their  individual  deficiencies,  and  an  individualized 
prescription  for  remedial  training.  In  this  briefing,  it  was  stressed  to 
each  team  that  they  would  be  retested  following  their  subsequent  patrol. 

At  the  completion  of  the  written  test,  a time  schedule  for  each  individual 
to  take  the  hands-on  performance  test  was  arranged. 


RESULTS  AN!)  DISCUSSION 


In  analyzing  the  data,  a separate  analysis  of  variance  (ANOVA)  was 
performed  on  each  of  the  individual  performance  tasks  and  on  each  of 
the  categories  in  the  written  test.  The  values  resulting  from  these 
ANOVAs  (F  values)  and  levels  of  significance  are  reported  in  Table  3. 

When  the  overall  ANOVA  for  a given  task  resulted  in  a significant  inter- 
action, additional  analyses  were  performed  on  the  data  to  determine 
simple  main  effects.  The  values  resulting  from  these  ANOVAs  are  presented 
in  Table  4. 

Performance  Test 


For  search,  contact  investigation,  tracking,  and  ping  interception,  the 
analyses  were  based  upon  the  percentage  of  front  panel  and  CRT  control 
switches  that  were  set  in  the  proper  position  according  to  operational 
doctrine.  In  the  case  of  SNR  determination,  however,  both  the  switch  set- 
ting and  the  answer  given  for  the  ratio  were  included  in  the  scoring.  At 
this  point,  three  additional  aspects  of  the  scoring  procedures  should  be 
pointed  out: 

1.  The  number  of  switches  scored  varied  across  tasks  since  only  those 
switches  that  were  relevant  to  performing  a particular  task  were  counted. 

2.  For  those  instances  where  a dependency  relationship  existed  between 
two  switches,  the  scoring  procedure  was  to  charge  only  one  error  if  the 
switches  were  set  at  improper  positions  but  in  proper  relationship  to  each 
other  for  the  setting  selected. 

3.  STs  who  made  no  attempt  to  perform  a given  task  on  the  pretest, 
contrary  to  test  instructions  to  do  so,  were  not  included  in  the  analysis 
for  that  task,  thus  the  group  N's  differ  across  tasks. 

In  discussing  the  performance  test,  the  results  on  the  primary  and 
auxiliary  uses  of  the  AN/B0R-20A  will  be  treated  separately.  The  primary 
uses  are  search,  contact  investigation,  and  tracking,  and  auxiliary  uses 
include  ping  interception  and  SNR  calculation.  Initially,  the  discussion 
will  focus  on  the  primary  uses. 
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Results  on  Primary  Uses 

There  are  differences  in  the  percentage  levels  of  both  pre-and  posttest 
performance  on  the  search,  contact  investigation,  and  tracking  tasks;  how- 
ever, as  shown  in  Figures  1,  2,  and  3,  there  is  a high  degree  of  consistency 
in  how  the  individual  treatment  groups  performed  across  these  three  tasks. 

On  all  three  tasks,  the  Training  Group  showed  significant  improvement  while 
the  Feedback  and  Control  Groups  showed  little  or  no  improvement.  For  all 
three  groups,  the  pretest  performance  on  these  tasks  was  uniformly  low. 

Over  all  groups  on  the  pretest,  only  about  75%  of  the  front  panel  and  CRT 
controls  were  set  properly  on  the  search  task,  65%  on  the  contact  investiga- 
tion task,  and  67%  on  the  tracking  task.  While  it  is  accurate  to  say  that, 
under  some  conditions,  all  of  the  available  contact  information  may  be 
obtained  even  when  certain  front  panel  controls  are  set  improperly,  the 
danger  is  that,  on  other  occasions,  these  improper  settings  may  make  it 
unlikely  that  vital  contact  information  will  be  obtained.  Further,  in  other 
more  serious  cases,  certain  settings  on  some  switches  make  it  impossible  to 
obtain  contact  information.  Additionally,  overdriving  the  signal,  a condition 
observed  frequently  during  testing,  may  create  spurious,  system-generated 
artifacts  that  can  mask  real  contact  information  or  be  misinterpreted  as 
relating  to  a nonexistent  contact. 

On  the  posttest,  the  Training  Group  improved  significantly  on  all  primary 
tasks.  The  group  set  92  percent  of  the  front  panel  and  CRT  controls  correctly 
on  the  search  task;  88  percent  on  the  contact  investigation  task;  and  80 
percent  on  the  tracking  task.  These  figures  represent  an  improvement  in 
performance  from  pretest  levels  of  16  percent  on  search,  22  percent  on  contact 
investigation,  and  11  percent  on  tracking.  The  fact  that  the  greatest  per- 
centage improvement  was  shown  on  the  contact  investigation  task  is  not 
surprising  in  view  of  the  fact  that  existing  operational  doctrine  on  this 
equipment  is  not  explicit  on  contact  investigation  procedures.  As  indicated 
previously,  since  there  was  no  existing  doctrine,  the  standard  adopted  for 
scoring  this  task  in  the  testing  program  was  based  upon  procedures  recommended 
by  job  experts  at  NISC.  These  procedures  were  also  incorporated  into  the 
training  materials.  Members  of  the  Training  Group  were  thus  provided  with 
training  documentation  in  an  area  where  none  existed  before.  The  lowest  per- 
centage of  improvement  was  shown  on  the  tracking  task.  On  this  task,  there 
may  have  been  less  motivation  to  learn  the  proper  procedures  due  to  many  of 
the  STs'  attitudes  toward  the  task.  On  the  pretest,  and  to  a lesser  degree 
on  the  posttest,  many  of  the  STs  were  somewhat  resistive  to  performing  this 
task.  A frequent  comment  was  that  on  the  job  they  were  never  required  to 
perform  tracking,  since  active  pursuit  of  a contact  is  not  the  primary  mission 
of  FBM  submarines. 

At  this  point,  a question  which  may  be  asked  is  whether  the  level  of  per- 
formance attained  by  the  Training  Group  on  the  posttest  is  adequate  for  these 
primary  tasks.  While  this  program  did  not  deal  specifically  with  the  determina- 
tion of  criterion  levels  for  performance,  it  is  safe  to  say  that  the  likelihood 
of  detecting  a contact  and  maintaining  a track  is  considerably  enhanced  when 
a greater  proportion  of  the  switches  on  the  control  panel  and  CRT  are  set 
properly.  In  this  regard,  it  should  be  pointed  out  that  the  improvement  shown 
by  the  Training  Group  was  achieved  by  spending  only  an  average  of  6.2  hours 
on  self-instruction  lessons,  2.8  hours  on  logic  trees,  and  1.8  hours  on  drills 
during  one  patrol.  It  seems  reasonable  to  expect  that  having  the  training 
materials  readily  available  for  refresher  training  could  result  in  additional 
Improvement  over  time. 
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Figure  3.  Pre-  and  Posttest  Performance  on  the  Tracking  Task. 
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Results  iiii  AuxtJ  I ary  list's 

I't.r  auxiliary  use's  of  Clio  AN/ IUJK-20A,  the  task  areas  of  ping  inter- 
ception anti  SNR  calculation  were  selected  for  testing.  As  shown  in  Figure  A, 
on  the  ping  interception  task  no  group  showed  significant  improvement  between 
pre-  and  posttesting,  although  the  trend  of  the  Training  Croup  was  in  the 
expected  direction.  In  view  of  the  relatively  high  performance  levels  on  the 
pretest,  failure  to  reach  significance  levels  of  improvement  can  probably  be 
attributed  to  a ceiling  effect.  As  shown,  the  average  pretest  performance 
level  for  the  three  groups  was  about  87  percent. 


PRE  POST 


Figure  4.  Pre-  and  Posttest  Performance  on  the  Ping  Interception  Task. 


* - 0n  t^le  SNR  determination  task,  the  improvement  between  pre-  and  post- 

testing was  better  for  all  groups  than  for  any  other  task.  On  the  posttest, 
all  three  groups  showed  significant  improvement  over  pretest  performance,  but 
the  groups  did  not  differ  statistically  from  each  other.  It  should  be  pointed 
out,  however,  that  pretest  performance  levels  were  quite  low,  as  shown  in 
Figure  5.  Results  on  the  SNR  task  are  an  exception  to  the  general  finding.  On 
all  of  the  other  tasks,  the  Control  and  Feedback  Groups  showed  little  or  no 
improvement  between  pre-  and  posttesting.  This  discrepant  finding  can  probably 
be  accounted  for  by  the  nature  of  the  SNR  task.  Since  SNR  can  be  calculated 
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at  a variety  of  front  panel  control  settings,  learning  to  do  this  task  is 
quite  simple,  even  though  many  STs  could  not  do  this  task  on  the  pretest. 

It  generally  involves  merely  setting  the  mode  selector  switch  to  the  proper 
position  and  determining  the  approximate  dB  value  from  the  CRT  presentation. 

In  view  of  the  simplicity  of  this  task,  it  seems  likely  that  learning  occurred 
from  exposure  to  the  pretest  and  this  contributed  to  the  improvement  shown  by 
all  groups. 


Written  Test 


There  were  two  categories  of  questions  included  in  the  written  test.  The 
first  category  contained  31  questions  relating  specifically  to  the  operation 
of  the  AN/BQR-20A.  In  general,  these  questions  covered  aspects  of  operation 
which  were  not  included  in  the  performance  test.  The  second  category,  con- 
sisting of  11  questions,  pertained  to  general  sonar  knowledges  which  an  operator 
should  know  to  properly  utilize  this  equipment.  When  the  written  test  was 
constructed,  it  was  anticipated  that  training  materials  would  be  developed 
to  cover  these  supporting  knowledges;  however,  due  to  time  and  personnel  con- 
straints it  was  not  possible  to  develop  such  training  materials  before  testing 
began.  Therefore,  the  two  categories  of  questions  were  separately  analyzed 
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because  the  training  materials  provided  instruction  on  the  former  category  of 
questions  but  not  on  the  latter.  On  the  equipment-related  questions,  pretest 
performance  by  all  groups  was  quite  similar,  as  shown  in  Figure  6.  On  the 
posttest,  however,  the  Training  Group  improved  significantly  (13%)  while  the 
Feedback  and  Control  Groups  showed  only  slight  improvement.  On  questions 
concerned  with  general  sonar  knowledge,  there  was  no  significant  difference 
between  groups  on  pre-  and  posttesting,  although  all  groups  did  show  slight 
Improvement  on  the  posttest.  Since  the  training  materials  covered  operation 
of  the  AN/BQR-20A  but  not  general  sonar  knowledges,  it  was  expected  that  the 
Training  Group  would  show  improvement  on  the  equipment-related  questions  but 
would  not  improve  appreciably  on  general  sonar  knowledges. 


Figure  6.  Pre-and  Posttest  Performance  on  the  Written  Test. 


It  should  be  noted  that,  because  of  the  overall  low  pretest  performance 
levels  on  both  the  written  and  performance  tests,  all  members  of  the  Training 
Group  were  assigned  most,  or  all,  of  the  remedial  training  materials.  Addi- 
tionally, in  most  cases  where  certain  units  of  training  were  not  assigned  to 
an  ST,  the  training  logs  indicated  that  he  completed  the  unassigned  units  on 
his  own  initiative.  Thus,  because  of  the  overall  need  for  training  in  the 


case  of  this  equipment,  the  value  of  diagnostic  feedback  was  not  as  evident 
as  it  might  be  in  other  applications  of  such  a program.  Ideally,  diagnostic 
feedback  should  provide  for  substantial  savings  in  training  time  and  effort  to 
the  individual  by  pointing  out  those  specific  areas  in  which  he  should  con- 
centrate his  training  efforts. 
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During  posttesting,  members  of  the  Training  Group  were  asked  to  complete 
a brief  questionnaire  in  which  they  evaluated  the  value  of  the  diagnostic 
feedback  information  and  the  training  materials.  On  a 10-point  scale,  with 
0 being  essentially  useless  and  10  being  very  helpful,  the  value  of  the  feed- 
back information  on  specific  deficiencies  received  an  overall  rating  of  7.4. 

On  the  same  10-point  scales,  the  self-instruction  lessons  received  an  average 
rating  of  7.5,  the  logic  trees  6.9,  and  the  drills  and  associated  tapes  6.0 
in  terms  of  how  helpful  these  materials  were  in  fulfilling  personal  needs 
for  training.  In  response  to  the  question,  "Do  you  feel  that  the  use  of  these 
training  materials  helped  you  to  better  understand  the  system  and  to  operate 
it  more  effectively?,"  all  of  the  STs  answered  yes  with  the  exception  of  two 
respondents  who  did  not  answer  the  question.  When  asked  whether  a program 
which  uses  diagnostic  tests  to  reveal  deficiencies  in  performance  and  pre- 
scribes remedial  training  should  be  implemented  for  other  systems  and  ratings, 
all  of  the  STs  answered  yes  except  for  one  who  did  not  answer  the  question. 

Objectivity  of  the  Performance  Test 

In  a testing  program  of  this  nature,  where  test  results  are  used  both  for 
diagnosing  performance  deficiencies  and  prescribing  remedial  training,  the 
degree  to  which  the  test  instruments  are  capable  of  yielding  the  same  decisions 
when  used  by  different  raters  is  a major  concern.  Consequently,  in  the  ST 
application,  the  objectivity  of  the  performance  test  was  evaluated  by  using 
two  observers  during  the  pretesting  of  13  STs  from  three  sonar  teams.  In 
the  12  problems  required  of  each  of  the  13  STs,  a total  of  1348  switch  positions 
were  observed  and  separately  recorded  by  each  rater.  Those  instances  in  which 
an  ST  did  not  attempt  a given  problem  were  not  included  in  evaluating  the 
results,  since  scoring  all  switches  incorrect  for  unattempted  problems  would 
result  in  an  inflated  value  of  agreement  between  raters.  Over  the  1348 
observations,  there  was  96  percent  agreement  between  raters  on  the  position 
of  the  controls.  This  resulted  in  a Phi  coefficient  of  .90.  Regarding  the 
4 percent  disagreement,  62  percent  of  this  amount  was  accounted  for  by  the 
CRT  controls  on  which  some  subjectivity  could  not  be  avoided.  An  additional 
12  percent  of  the  disagreements  resulted  from  one  or  the  other  observer  failing 
to  mark  a given  control  position  on  the  scoring  sheet.  The  high  percentage  of 
agreement  between  the  two  raters  provides  evidence  that  this  test  is  suf- 
ficiently objective  to  yield  essentially  the  same  decision  when  used  by  various 
raters. 

Reliability  of  the  Tests 

A rough  check  on  the  reliability  of  the  test  instruments  was  performed  by 
calculating  the  correlations  between  pre-  and  posttest  results  on  both  the 
written  and  performance  tests  for  the  Control  Group.  The  coefficients  for  both 
the  written  test  (_r  = .88,  n = 16;  £ < .001)  and  the  performance  test  (_r  = .61, 
ri  = 16,  £ < .01)  were  significant.  In  view  of  the  fact  that  the  pre-  and 
posttests  were  administered  at  least  5 months  apart,  these  coefficients  provide 
evidence  that  the  tests  have  a considerable  degree  of  stability  over  time. 
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CONCLUSIONS 


The  results  from  the  12  ST  teams  tested  indicate  that,  in  the  FBM  fleet, 
there  are  serious  overall  deficiencies  in  the  operation  of  the  AN/BQR-20A. 

The  training  packages  which  were  developed  appear  to  he  capable  of  correcting 
these  deficiencies.  It  should  be  remembered  that  the  considerable  improve- 
ment shown  by  the  Training  Group  was  the  result  of  an  average  of  less  than 
11  hours  of  training  during  one  patrol.  It  is  likely  that  more  extensive 
use  of  the  materials  will  result  in  additional  improvement. 

In  this  application  of  the  Personnel  Readiness  Training  program,  feedback 
in  the  form  of  specific  performance  deficiencies  was  not  in  itself  sufficient 
to  bring  about  improvement  in  job  performance.  This  could  be  due  to  a lack 
of  AN/BQR-20A  training  materials  in  the  Fleet.  During  the  course  of  this 
project,  it  was  found  that  existing  documentation,  while  technically  accurate, 
is  inadequate  and  poorly  organized  from  the  training  standpoint.  Apparently, 
members  of  the  Feedback  Group  believed  the  diagnostic  information  they  were 
given  was  helpful;  on  the  posttest  debriefing  questionnaire,  STs  in  the  Feed- 
back Group  rated  the  value  of  the  diagnostic  feedback  at  7.4  on  a 10-point 
scale  with  10  being  very  helpful.  Nevertheless,  this  group  showed  little  or 
no  improvement  on  most  tasks.  It  could  be  that,  even  though  feedback  informa- 
tion was  considered  beneficial,  the  STs  found  it  difficult  or  impossible  to 
overcome  their  deficiencies  through  existing  documentation.  If  appropriate 
training  materials  had  been  available,  their  performance  might  have  shown 
greater  improvement  on  the  posttest. 
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Because  operator  deficiencies  seem  to  be  very  widespread  in  the  KBM 
fleet,  it  appears  that  the  diagnostic  testing/shipboard  training  program 
should  be  implemented  in  two  phases,  starting  with  a shipboard  training 
phase  and  then  incorporating  a diagnostic  testing  phase.  Since  the  diag- 
nostic testing  resulted  in  all  of  the  STs  being  assigned  most,  or  all,  of  the 
training  materials,  initiation  of  AN/BQR-20A  diagnostic  testing  in  the  FBM 
fleet  at  this  time  would  appear  to  be  an  inefficient  solution  to  the  problem. 
Therefore,  because  of  existing  low  performance  levels,  it  would  appear  to  be 
more  efficient  to  carry  out  implementation  in  two  phases.  In  the  initial 
phase,  one  set  of  the  shipboard  remedial  training  materials  would  be  dis- 
tributed to  the  Sonar  Division  of  each  crew  on  all  FBM  submarines.  This  would 
provide  adequate  training  documentation  which  presently  does  not  exist  on  this 
equipment.  In  the  second  phase  of  implementation,  which  should  occur 
approximately  6 months  after  distribution  of  the  training  materials,  diagnostic 
testing  would  be  initiated  to  determine  fleet  performance  levels  after  exposure 
to  the  training  materials.  Refresher  training  from  the  shipboard  training 
package  would  be  prescribed  on  the  basis  of  test  results.  If,  after  testing 
a sample  of  FBM  ST  teams,  it  is  found  that  performance  levels  have  improved 
to  an  adequate  level,  the  diagnostic  testing  could  be  terminated.  If,  on  the 
other  hand,  performance  levels  of  a significant  number  of  STs  are  still  found 
to  be  inadequate,  the  diagnostic  testing  and  assignment  of  remedial  training 
should  be  continued.  The  testing  could  be  incorporated  into  an  ongoing  program 
such  as  Operational  Readiness  Inspections. 

For  the  initial  phase,  two  sets  of  training  materials  consisting  of  self- 
instructional  units,  logic  trees,  drills,  and  training  tapes  would  be  required 
for  each  submarine.  The  estimated  cost  of  reproducing  the  two  sets  of  materials 
would  be  $30.00  per  submarine.  For  the  second  phase,  the  diagnostic  testing 
program  would  require  a highly  qualified  AN/BQR-20A  operator  to  administer  the 
test.  He  should  be  an  ST  of  E-7  pay  grade  or  higher.  One  AN/BQR-20A  would  be 
required  to  conduct  the  performance  test.  Since  this  equipment  is  presently 
installed  at  most  submarine  shore  facilities,  arrangements  for  time-sharing 
could  probably  be  arranged  at  the  Fleet  Training  Centers  or  Sonar  Information 
Centers.  If  a dedicated  AN/BQR-20A  is  required,  however,  the  approximate  cost 
of  a new  system  is  $22,500.  The  only  additional  requirement  would  be  a class- 
room for  testing  purposes. 
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